Global Address Space, Non-Uniform Bandwidth: A Memory System Performance Characterization of Parallel Systems

Many parallel systems offer a simple view of memory: all storage cells are addressed uniformly. Despite a uniform view of the memory, the machines differ significantly in their memory system performance (and may offer slightly different consistency models). Cached and local memory accesses are much faster than remote read accesses to data generated by another processor or remote write to data intentionally pushed to memories close to another processor. The bandwidth from/to cache and local memory can be an order of magnitude (or more) higher than the bandwidth to/from remote memory. The situation is further complicated by the heavy influence of the access pattern (i.e. the spatial locality of reference) on both the local and the remote memory system bandwidth. In these modern machines, a compiler for a parallel system is faced with a number of options to accomplish a data transfer most efficiently. The decision for the best option requires a cost benefit model, obtained in an empirically evaluation of the memory system performance. We evaluate three DEC Alpha based parallel systems, to demonstrate the practicality of this approach. The common DEC-Alpha processor architecture facilitates a direct comparison of memory system performance. These systems are the DEC 8400, the Cray T3D, and the Cray T3E. The three systems differ in their clock speed, their scalability and in the amount of coherency they provide. The DEC 8400 is a shared memory, symmetric multiprocessor based on a high speed bus offering sequential consistency; the Cray T3D and T3E are scalable multicomputers based on a scalable 3D torus interconnect and either do not cache remote accessesat all (T3E) or provide only partial memory consistency within a node (T3D) and therefore typically leave consistency to the application or compiler. Our performance characterization shows that although the clock rate of the DEC 8400 doubled compared to the Cray T3D, the DEC 8400 offers only modest improvements in the performance of remote memory operations over the Cray T3D. The local and remote memory system performance of the Cray T3E This research was sponsored in part by the Advanced Research Projects Agency (ITO) monitoredby SPAWAR undercontract N00039-93-C-0152. T. Stricker’s current address: Institut für Computer Systeme, ETH Zürich, Switzerland. Copyright 1997 IEEE. Published in the Proceedings of the THird International Symposium on High Performance Computer Architecture, February 1-5, 1997 in San Antonio, Texas, USA. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the IEEE. Contact: Manager, Copyrights and Permissions / IEEE Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. Telephone: + Intl. 908-562-3966. matches the doubled clock speed of the processor.